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A Viterbi decoder’s performance loss due to quantizing data from the additive 
white Gaussian noise (AWGN) channel is studied. An optimal quantization scheme 
and branch metric calculation method are presented. The uniformly quantized 
channel capacity C u {q) is used to determine the smallest number of quantization 
bits q that does not cause a significant loss. The quantizer stepsize which maximizes 
C u (q) almost minimizes the decoder bit error rate (BER). However, a slightly larger 
stepsize is better, like the value that minimizes the Bhattacharyya bound. The range 
and renormalization of state metrics are analyzed, in particular for I< — 15 decoders 
such as the Big Viterbi Decoder (BVD) for the Galileo mission. These results are 
required to design reduced hardware complexity Viterbi decoders with a negligible 
quantization loss. 


I. Introduction 

Theoretically, Viterbi decoding is a maximum- 
likelihood decoding algorithm for convolutional codes. In 
practice, the main performance loss results from quantiz- 
ing input data with q bits. The decoder’s hardware com- 
plexity and speed depend strongly upon q and the state 
metric register length £. Therefore, these parameters must 
be chosen as the smallest values that do not cause a signif- 
icant bit signal-to-noise ratio (E^/No) loss. A constraint 
length K = 15 decoder performs double the computation 
of a K — 14 decoder, but requires about 0.1 dB less Ei/Nq 
for a bit error rate (BER) of 0.005. Since part of the de- 
coder’s hardware complexity increases only linearly with q , 
even a 0.01-dB quantization loss is large. However, given 


that one must construct a fully parallel K = 15 (or K = 7) 
decoder, a slightly larger loss might be acceptable or re- 
quired by hardware and speed constraints. 

The uniformly quantized, additive white Gaussian 
noise (AWGN) channel capacity C u (q) is used to estimate 
the quantizer stepsize A and smallest q that result in a 
negligible loss. For each q , almost minimum BER occurs 
when A maximizes C u (q ) or minimizes the Bhattacharyya 
bound 7 . New methods are presented to minimize the 
state metric register length l in bits. These estimates 
are verified by simulations of three codes : the constraint 
length K = 7, rate R = 1/2, NASA standard code; the new 
experimental K = 15, R = 1/4, Galileo code [1]; and the 
K- 15, R = 1/6, “2-dB” code [2]. 
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These results are used to determine the best design 
parameters q , and A for K — 15, rate 1/4 and rate 1/6 

decoders. Using 6 quantization bits with 10-bit state met- 
ric registers would substantially reduce the Big Viterbi De- 
coder (BVD) [4] hardware complexity and allow the sys- 
tem clock frequency to decrease by a factor of 0.56 as com- 
pared to the current design, which has q — 8 and £= 16. 
Using q = 5 would cause an E^/Nq loss of 0.02 dB at a BER 
of 0.005 for the Galileo code or “2-dB” code, but there is 
no measurable Eb/No loss when q = 6. Since the same 
losses occurred for 8-bit symbol error rates (SER), these 
results apply when an outer block code is concatenated 
with the convolutional code. 


II. Branch Metrics 

When an encoded 0 or 1 is mapped to -f-1 or — 1, 
respectively, and then transmitted, the receiver’s demod- 
ulator output is a conditionally Gaussian random vari- 
able y with mean +m or and the same variance 

<r 2 — m 2 /(2REb/No) as the zero-mean AWGN channel 
noise. (This holds for binary phase shift-keyed [BPSK] 
signaling with ideal coherent detection.) 

For the AWGN channel, a Viterbi decoder finds the 
trellis path with minimum Euclidean distance (or equiv- 
alently, minimum negative inner product) to the received 
sequence. Thus, each trellis branch metric is the inner 
product of the length n branch label (with 0 and 1 re- 
placed with -fl and —1) and the negative of a received 
vector [2/1 , V 2 > * • • > y n ]* Hence, the decoder adds —y t - or 
+y; (equivalently ( — y* + |y,|)/2 or (y, + |y,|)/2 when a is 
fixed, because incrementing or multiplying all branch met- 
rics by a constant does not change the decoder’s output) 
to the metrics of those branches with a -|-1 or —1 in posi- 
tion i. Therefore, the decoder may add |t/j| to the metrics 
of branches having different signs in position ? than that 
of yi , and zero otherwise. This sign-magnitude method is 
used throughout this article because it halves the branch 
and state metric maximum ranges, as compared to using 
standard integer metrics [3,4]. For example, using this 
method in the Scarce-State K — 7, rate 1/2 decoder [5] 
would substantially decrease the chip circuitry. 


III. Quantization 

When zeros and ones are equally likely in the encoder 
input data, 

Pr(|y| - x )~\ [ Pr (y = *!+!) + Pr(y = -* I +1)] 


+ \ [iMy = * I -1) + p r(y = ~ x I - 1 )] 

= Pr(y — x | +1) + Pr(y = x | -1) 


y/2 7T <7 [ 


,-(r-m) 3 /2<7 3 e -(zr + m) 3 /2<7 2 


In this article, m — 0.84 volts. The probability distribu- 
tion function of |y| (Fig. 1) suggests that more quantiza- 
tion levels are required for the K = 15 codes operating 
near 0 dB (high noise variance) than the NASA code at 
E b /N 0 = 2.25 dB. 


Let the random variable J be the quantized value of 
y and for — 2 q ~ l — 2 < j < 2 q ~ l — 2 define 


pj = Pr(J - j | +1) 


0*+0.5)A 

1 f -(t/-m) 3 /2<7 2 

J 

O’- 0.5) A 


dy 


For j — ±(2 9_1 — 1), pj is the above integral with 
limits (j — 0.5) A and +oo, or — oo and (j + 0.5) A. 

Since |Jj|, . . . , |J n | are summed to form branch met- 
rics, the absolute error [</» — y, | in quantizing yi is also the 
contribution to the branch metric error incurred. A de- 
coder using signed integers to represent could conceptu- 
ally use 0, ±A, ±2A, . . . , ±(2 q ~ x - 1 )A for any real num- 
ber A, because multiplying all metrics by A has no effect. 
Therefore, the quantizer thresholds should be uniformly 
spaced A volts apart at ±A/2, ±3A/2, , . . , dt(2 9_1 )A/ 2, 
because this minimizes the metric error defined above (and 
also any positive function of J, — y,). Thus, only uniform 
quantization schemes, characterized by q and A, are con- 
sidered herein. (Several simulations of the NASA code 
using 3-bit integer branch metrics and nonuniform quan- 
tization schemes never produced lower BERs than using 
the best A). 

For q — 3, Ji is normally one of 7 values from —3 to 
-1-3, so quantizer levels +4 and —4 are appended (Fig. 2) to 
decrease the BER near that for 8 levels and standard inte- 
ger metrics. Thus, the maximum magnitude of J, 2 9-1 — 1, 
will be replaced by 4 instead of 3 for <7 = 3 throughout this 
article. In rate 1/2 decoders, a branch metric of 8 is de- 
creased to 7 so that q = 3 bits still represent all possible 
values. Since Pr(|J,| = j) = Pj -fp-j, this event occurs 
with probability (p+4 + P-4) 2 , which is only 0.11 for the 
NASA code at E b /N 0 = 2.25 dB. 
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IV. Quantizer Stepsize 

Ideally, the uniform quantizer stepsize A should min- 
imize BER and SER over the decoder’s operating range of 
channel noise levels. In practice, a A which almost min- 
imizes the BER for the lowest expected Eb/No will also 
nearly minimize both the BER and SER when Eb/No in- 
creases by up to 1 dB. Simulations (described later) indi- 
cate that the A that maximizes channel capacity is near 
optimum. 


The A that minimizes 7 is the safest choice because it 
is slightly larger than the stepsize which minimizes BER. 
Also, minimizing 7 yields the lowest BER for q — 3 with 
9 quantizer levels (Fig. 4). Finally, the corresponding 8- 
bit SER curves are not shown because they have the same 
relative shape and spacing as the BER curves in Fig. 4. 
Many sets of software simulations were run for the NASA 
code and the Galileo code. The values of q were 3, 4, 5, or 
6 and Eb/No ranged from 0 dB to 3.5 dB. 


Since the binary-input quantized AWGN channel is 
symmetric, capacity is achieved with equiprobable inputs: 


2* -1 — 1 

Ct i(q) = I + 1 ) lo S2 

j=- 2«~ 1 +l 


2 pjj | +1) 

.pO' I +1) + p(j | -1). 


= 1 - J2 Pi lo S2 



bits per channel use. Figure 3 shows how rapidly the max- 
imum possible uniformly quantized AWGN channel capac- 
ity C u (q) approaches its limit for several noise variances; 
C u (3) is based upon the 9-level quantizer in Fig, 2 instead 
of using 8 or 7 levels. A^ = 4or^ = 5 quantizer has 
15 or 31 levels, respectively. The data points in Fig. 2 
indicate C u (q ) for integer values of q. The lines between 
data points are channel capacities when uniform quantiz- 
ers have intermediate numbers of levels, such as 24. 

The curves in Fig. 3 show that there is negligible 
capacity gain for q > 6, and in fact C u {S)/C u (oo) > 0.9975 
suggests that there will be a very small loss for q = 5. 
Figure 4 shows how the performance of the NASA code 
at Eb/No = 2.25 dB varies with q and A. Observe that 
the minimum BER for q = 5, 4, and 3 increases roughly in 
proportion with the decrease in capacity. Also, for q — 5, 
there is a negligible loss and the BER increases extremely 
slowly for A greater than the optimum. Therefore, for q > 
4, it is important to choose A larger instead of smaller than 
the best value. The labels C and 7 in Fig. 4 indicate the 
stepsizes that respectively maximize C u (g) and minimize 
the Battacharyya bound parameter 


2*" 1 — 1 

7 = 53 vpJp^ 

j — — 2 g ~ 1 -f 1 

which is a measure of the channel noise level: near 0 for 
high Eb/No and approaching 1 for very noisy channels. 


In all simulations, the As which maximize C u (q) or 
minimize 7 were, respectively, slightly smaller or larger 
than the A that minimized BER. For q = 3 or 4, the As 
which minimize the quantizer mean-square error or abso- 
lute error were too large. 

The simulations in Fig. 5 for the K — 15 codes show 
that using q = 5 or 4 costs 0.02 dB or 0.05 dB at the 
BER of 0.005 required for images. These Eb/No quantiza- 
tion losses are the same when the Viterbi decoder output 
becomes the input to an outer block decoder, because the 
8-bit symbol error rate curves are spaced the same distance 
apart as the BER curves. In all simulations, the uniform 
spacing A was chosen to minimize 7. 


V. State Metric Renormalization 

For each received n-vector and encoder state, a 
Viterbi decoder finds the trellis path with least total 
branch metrics into the state. Since the state metrics are 
stored in ^-bit registers, occasionally they must all be de- 
creased to avoid overflow. This renormalization can be ac- 
complished by zeroing every register’s most significant bit 
(msb), which is equivalent to subtracting 2* _1 from every 
metric if all registers have msb = 1. However, detecting 
when all 2 K ~ l metrics simultaneously have msb = 1 is 
impractical for a K = 15 decoder such as the BVD. 

At each trellis level, let the random variable M be 
the difference between the maximum and minimum state 
metrics. If any state metric is > 2 /-1 -f 2 l ~ 2 , (its two 
most significant bits are 1) and M < 2* -2 , then all met- 
rics are > 2 /_1 , so every msb = 1. In the BVD, £ — 16 
was chosen to guarantee that 2* -2 > M, and so a sin- 
gle state metric is monitored and renormalization occurs 
when the two most significant bits are 1. The following 
improved method should be used when £ is reduced so 
that M > 2*~ 2 . Let W be the maximum of the metrics of 
the all-zeros state, the all-ones state, and the state with a 
one input followed by K — 2 zeros. Since most state met- 
rics differ from one of these three metrics by only a few 
|«/*| contributions, W is close to the largest state metric 
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(Galileo code simulations verified this). Therefore, renor- 
malization could occur when W exceeds a threshold such 
as 2 / " 1 -f 2 l ~ 2 -h 2*~ 3 . If more metrics are monitored, the 
threshold can be set closer to — 1 because W will be 
closer to the largest state metric. 

Definition. Let D be the maximum, over all nonzero 
states s, of the least-weight trellis path from the all-zeros 
state into state s. 

Lemma. M < D[2 q ~ l — 1) 

Proof. Let b and w be the states with lowest and 
highest metrics. Since a convolutional code is linear, there 
exist two trellis paths from some state c, one into state b 
and the other into state w } whose branch labels differ in 
D or fewer positions. Since the maximum contribution to 
a branch metric by one J; is 2 ?_1 - 1, the state metric of 
w is at most the state metric of 6 plus D(2 q ~ l — 1). 

Corollary. In the absence of noise, 

M = Mq = D ■ j m 

where j m = [0.5 + m/AJ is the quantizer output when 
+m volts is input. 


where the subscript i denotes the coefficient of x l in the 
polynomial within the braces. 

Proof. Let b and w be the states with lowest and 
highest metric. An upper bound on Pr(A/ = f) is obtained 
by considering the worst possible case: the survivor path 
for state w differs from that for state b in exactly D po- 
sitions, and in these positions, the survivor path branch 
labels of w have a different sign than the received J t *. Then 
Pr (M = 2) is the coefficient of x % in [m(a:)] . 

To achieve a particular (very low) probability, t must 
be unrealistically large since the above bound is not very 
tight. This is fine, because t could be chosen as the least 
power of 2 such that Pr (M > t) < 10 -5 . Then setting 
£ = 1 -h log 2 t results in no loss of performance. 

The current BVD design has q = 8 and £ — 16 to ac- 
commodate M < n(K — 1)(2^" 1 — 1) and two extra bits for 
renormalization. This results in full maximum-likelihood 
decoder performance. However, using g = 6, A = 0.14, and 
£=10 for the BVD operating at 0 to 1 dB E^/Nq would 
not increase the BER or SER detectably, but would reduce 
the decoder hardware. Furthermore, the system clock fre- 
quency and thus timing constraints would be reduced by 
a factor of 10/18. 


For nonsystematic codes, D is near df ree and usually 
much less than n(K — 1), the maximum possible. The 
NASA code has d free = 10, D — 8, and n(I\ — 1) = 12. 
Since D = 33 for the Galileo code, Mq = 132 for q = 5, 
A = 0.20, and m = 0.84. Since D — 50 for the rate 1/6 
“2-dB” code, Mq = 200. Simulations for the Galileo and 
NASA codes show that Mq is an upper bound on the mean 
of M when the channel is noisy and 2A/ 0 is always greater 
than M, 

As in the q = 3 case where levels +4 and —4 were 
adjoined, a rule for limiting branch metrics may be derived 
by computing their probabilities. Define 


2 q ~ 1 — 1 

m(x) = p 0 + ^2 (Pj + P-j ) xl 
j = 1 

Then Pr(|J*[ = j) = {m(z)}-, the coefficient of in 
m(x ). Since the largest possible branch metric is the sum 
of n independent values | J*|, . . . , |J n |, it equals t with prob- 
ability {[ m (2 r )] n }f* Thus A f could be reduced by limiting 
branch metrics. 


D(2 q ~ l — 1 ) 

Claim. Pr(Af > f) < ^ 

i=t 1 


When £ < 1 + log 2 [D(2 <? "" 1 - 1)], then occasionally a 
state metric may overflow, whereupon it is immediately 
decreased by 2*, instead of 2 /_1 at the next renormaliza- 
tion. Protecting against overflow is important because a 
state with a high metric might suddenly become one of the 
best states, causing the decoder to make wrong decisions. 
This can be avoided by setting state metrics that overflow 
equal to all ones (2* — 1). Then states with very high met- 
rics remain this way even after renormalization so they do 
not affect the decoder’s output. An underflow is the event 
that occurs at renormalization when a state metric has 
msb = 0, in which case the metric is effectively increased 
by 2 / ~ 1 . Rarely, underflows may occur because it is infea- 
sible to continuously check all 2 K ~ l state metrics to find 
the least value. In conclusion, overflows can be prevented 
by extra hardware, but underflows will occasionally hap- 
pen. In practice, always examining several state metrics 
gives a good approximation of the current metric size and 
range M. Hence, renormalization can take place so that 
overflows and underflows occur with very low probability. 

Myth. When state metrics overflow or underflow, 
the decoder fails completely. 

One million decoded bit simulations for q = 5 and 4 
with short state metric registers having £ = 9 and 8 bits, 
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respectively, yielded the same results as in Fig. 5, because 
the odd underflow or overflow that occurred did not signif- 
icantly affect the output. This follows from the Viterbi de- 
coder’s robustness and tolerance of occasional state metric 
disruptions. Further shortening of the state metric regis- 


ters to 8 and 7 bits resulted in a graceful BER increase, 
as though q was being decreased. This behavior is ex- 
pected because the overall trellis path metric resolution 
is the decoder parameter, affected by input quantization, 
which influences decisions. 
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Fig. 1. Received signal magnitude distribution. 
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Fig. 3. Uniformly quantized AWGN channel capacities. 


Fig. 2. Optimal quantization for 3-bit branch metrics. 
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Fig. 5. K = 15 code simulations. 


Fig. 4. Performance of the NASA code on the uniformly 
quantized AWGN channel. 







